[NPUW]Optimize MoE (GPT-OSS-20B) TPS on NPU - DEVICE_ROUTED.#33847
Open
intelgaoxiong wants to merge 3 commits intoopenvinotoolkit:masterfrom
Open
[NPUW]Optimize MoE (GPT-OSS-20B) TPS on NPU - DEVICE_ROUTED.#33847intelgaoxiong wants to merge 3 commits intoopenvinotoolkit:masterfrom
intelgaoxiong wants to merge 3 commits intoopenvinotoolkit:masterfrom
Conversation
github-actions
bot
added
category: build
intelgaoxiong
force-pushed
the
xiong/gpt-oss_device_routed
branch
2 times, most recently
from
d8f7978 to
10e6b84
Compare
intelgaoxiong
changed the title
intelgaoxiong
force-pushed
the
xiong/gpt-oss_device_routed
branch
from
10e6b84 to
97b9ea3
Compare
intelgaoxiong
force-pushed
the
xiong/gpt-oss_device_routed
branch
3 times, most recently
from
3dacd25 to
270410d
Compare
Contributor
Author
intelgaoxiong
force-pushed
the
xiong/gpt-oss_device_routed
branch
from
270410d to
7099966
Compare
Convert gather to 2D. Gather before convert. Keep gather indices as constant. Use JustInferRequest for DEVICE_ROUTED mode. Clean up transformations for DEVICE_ROUTED. Update config for DEVICE_ROUTED: BEST_PERF + not cut LM head. Refactor device routed transformation. Refactor GatherTo2DGather. Apply MoE defaults if not explicitly set in external config. Collect MoE nodes in single loop. Signed-off-by: intelgaoxiong <xiong.gao@intel.com>
multiply considered. Signed-off-by: intelgaoxiong <xiong.gao@intel.com>
Signed-off-by: intelgaoxiong <xiong.gao@intel.com>
intelgaoxiong
force-pushed
the
xiong/gpt-oss_device_routed
branch
from
7099966 to
6fbaa79
Compare
Contributor
Author
|
Rebased. |
AlexanderKalistratov
approved these changes
Feb 6, 2026
Contributor
AlexanderKalistratov
left a comment
AlexanderKalistratov
left a comment
There was a problem hiding this comment.
idk looks fine for me.
But please wait review of others
| std::dynamic_pointer_cast<ov::op::v0::Constant>(tile->input_value(1).get_node_shared_ptr()); | ||
| if (repeats_const) { | ||
| auto repeats_data = repeats_const->cast_vector<int64_t>(); | ||
| if (!repeats_data.empty() && repeats_data[0] > k_value) { |
Contributor
There was a problem hiding this comment.
So it is a possible situation that repeats_data[0] <= k_value?
| } else { | ||
| // Constant reshape - check if dim 0 is expert dimension | ||
| auto shape_data = shape_const->cast_vector<int64_t>(); | ||
| if (nodes.num_experts > 0 && !shape_data.empty() && |
Contributor
There was a problem hiding this comment.
nodes.num_experts > 0 implicitly assumes that we find Tile node first.
| } | ||
| } | ||
|
|
||
| return nodes; |
Contributor
There was a problem hiding this comment.
So the whole function relies on some unobvious assumptions and layer names.
Why did you prefer it over MatcherPass and pattern matching?
| } | ||
| } | ||
|
|
||
| void transform_dynamic_reshapes(LayerNodes& nodes) { |
Contributor
There was a problem hiding this comment.
Why are we doing this?
Does it helps us later?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
Background:
#33372 implemented
HOST_ROUTEDprocessing for MoE decoding.But the trivial submission overhead limits the decoding throughput.
Optimization:
This PR optimized MoE TPS with
DEVICE_ROUTEDprocessing:Gatheroperations, avoiding graph splitting and reducing host-device overhead.TPS can be improved from 12 t/s to 17.9 t/s.
NPUW config:
Tickets: